Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning.

Additional Feedback: Line-by-line comments: Figure 1 - This figure is quite cluttered. I would recommend removing/simplifying some of the graphics (e.g. the thought bubbles) and, when possible, moving them outside the environment canvas. Line 37 - Do the citations on lines 38-39 substantiate the preceding claim that language actually influences children's exploration behavior? This seems like a very difficult claim to test, how do you disentangle mental maturity with language acquisition? Note that I do not consider children "narrating their ongoing activities" to be a meaningful change in behavior if it is not accompanied by a change in how the children actually complete those activities.

Reviewers agreed that the paper proposes an interesting model for learning language conditioned goal reaching policies and performs a thorough investigation on a simple task. There was agreement that the environment studied in the paper is quite simplistic and that the paper would benefit from a task with a richer grammar/goal space. Nevertheless, the results on the task in the paper are sufficiently interesting for acceptance as a poster.

Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning.

Autonomous reinforcement learning agents must be intrinsically motivated to explore their environment, discover potential goals, represent them and learn how to achieve them. As children do the same, they benefit from exposure to language, using it to formulate goals and imagine new ones as they learn their meaning. In our proposed learning architecture (IMAGINE), the agent freely explores its environment and turns natural language descriptions of interesting interactions from a social partner into potential goals. IMAGINE learns to represent goals by jointly learning a language model and a goal-conditioned reward function. Just like humans, our agent uses language compositionality to generate new goals by composing known ones. Leveraging modular model architectures based on Deep Sets and gated-attention mechanisms, IMAGINE autonomously builds a repertoire of behaviors and shows good zero-shot generalization properties for various types of generalization. When imagining its own goals, the agent leverages zero-shot generalization of the reward function to further train on imagined goals and refine its behavior. We present experiments in a simulated domain where the agent interacts with procedurally generated scenes containing objects of various types and colors, discovers goals, imagines others and learns to achieve them.